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Abstract 

For Gaussian sampling matrices, we provide bounds on the minimal number of 
measurements m required to achieve robust weighted sparse recovery guarantees in 
terms of how well a given prior model for the sparsity support aligns with the true 
underlying support. Our main contribution is that for a sparse vector x G sup¬ 
ported on an unknown set 5 C {1,, N} with |5| < k, if S has weighted eardinality 
•= weights on 5° exhibit mild growth, ojj > 7 log(j/a;(5)) 

for j G S‘^ and 7 > 0, then the sample complexity for sparse recovery via weighted £ 1 - 
minimization using weights loj is linear in the weighted sparsity level, m = 0{uj{S) /”f). 

This main result is a generalization of special cases including a) the standard sparse 
recovery setting where all weights ojj = 1, and m = O {k log (N/k)); b) the setting 
where the support is known a priori, and m = 0{k); and c) the setting of sparse 
recovery with prior information, and m depends on how well the weights are aligned 
with the support set S. We further extend the results in case c) to the setting of 
additive noise. Our results are nonuniform that is they apply for a fixed support, 
unknown a priori, and the weights on S do not all have to be smaller than the weights 
on 5° for our recovery results to hold. 

Keywords, weighted sparsity, sample complexity, compressive sensing, Gaussian width, 
weighted .^i-minimization 

1 Introduction 

Consider the scenario where we would like to acquire an unknown vector x G with 
assumed sparse or compressible representation from m <C X linear measurements of the 

form Hi = (aj, x). We are also interested in the noisy setting where instead yi = (a*, x) -1-6* 
th 

where e* is the component of a noise vector e with noise level ||e||2 < rj- Recall 
that X G is said to be /c-sparse if it is supported on a subset S C {1,2,... ,N} of 
cardinality |5| < k. It is well known by now that for an unknown /c-sparse vector x, given 
a measurement matrix A G , m = 0{k \og{N/k)) noisy random linear measurements 

y = Ax -|- e suffice to guarantee stable and robust reconstruction of x via the following 
t'l-minimization problem: 

x^ := arg min ||z||i subject to || Az — y ||2 < r/, (LI) 
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with the solution having an approximation error of the form ||x^—x ||2 < Ci ||x—Xfc|| 2 +C' 2 r/, 
where x^ is the best fc-term approximation to x. In fact, the number of measurements 
m > Ck\og{N/k) is order-optimal - given no additional information about the support of 
X, no algorithm can provide a better worst-case recovery guarantee using fewer measure¬ 
ments. This forms the basis of the field of compressive sensing [Don06,CRT06,FR13]. 

In many practical problems of interest, one might have additional prior information 
about which indices within the full index set [A^] = {1,2,...,A^} are more likely to appear 
in the best /c-term approximation. So in the noisy setting above, given a prior distribution 
of weights u!j > 0 associated to the indices, a natural choice of reconstruction algorithm 
is weighted t'l-minimization: 

X := arg min ||z||^i subject to || Az — y ||2 < ?/, (WLl) 

where ||z||t^^i = with larger-magnitude weights (or, perhaps more precisely, 

larger-magnitude “penalties”) Uj > 0 are assigned to indices believed less likely to con¬ 
tribute to the support of x. In case the prior distribution of weights is in fact aligned with 
the sparsity structure of x, it is natural to ask whether weighted t'l-minimization might 
have reduced sample complexity compared to unweighted .^i-minimization, (LI). This 
has been the subject of several works in the area of compressive sensing, see for exam¬ 
ple [VBMP07, CWB08, KXAH09, VLIO, JaclO, XKAHIO, MYll, FMSY12, OKH12, MY12, 
RW15,MS14]. 

Existing results show better sparse reconstruction results for weighted t'l-minimization 
than the standard unweighted .^i-minimization, which is usually demonstrated in the form 
of weaker recovery conditions that guarantee recovery with reduced sample complexi¬ 
ties [KXAH09, RW15, MS14]. In the extreme case, one might know the support of x 
exactly; say, for example, S = {1,2,... ,k}. In the noiseless setting, x can then be exactly 
recovered from y using e.g. m = k Gaussian random measurements, with probability 1, 
the measurements y = Ax = A 5 X 5 will be invertible. In the middle ground between 
knowing the support set a priori and knowing only that the support is some subset of [N] 
of cardinality k, what is the sample complexity of sparse recovery? Clearly the answer lies 
between the extremes of m = 0{k) and m = 0{k\og{N/k)). The purpose of this paper is 
to make the sample complexity precise, in terms of the distribution of the weights and how 
well this distribution aligns with the true underlying support of the signals of interest. 

There are several motivating examples in which one naturally is faced with a sparse 
recovery problem and a prior distribution over the support. For example, the smoothness 
of a function corresponds to the rapid decay of its Fourier series, so lower-order frequency 
basis functions are more likely to contribute to the best s-term approximation, hence 
making smoothness a structured sparsity constraint. It has been demonstrated in [RW15] 
that such structured sparse expansions are imposed by weighted fp coefficient spaces in 
the range 0 < p < 1 , and that weighted £i-minimization can be used as a convex surrogate 
for weighted .^p-minimization with p <1. 

Alternatively, the recursive recovery of time sequences of sparse signals when their sup¬ 
ports evolves slowly over time [JaclO] is another setting with prior information about the 
support. In this case the support from the previous time can be used as a support estimate 
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in the current time. This can be applied to real-time dynamic MRI reconstruction, real¬ 
time single-pixel camera video imaging. More applications where sparse reconstruction 
for time sequences of signals/images may be needed can be found in [VLIO]. 

Similarly in medical imaging, natural images have tree-structured wavelet expansions 
and are less sparse on lower-order wavelets. Precisely, wavelet functions act as local 
discontinuity detectors, and a signal discontinuity will give rise to a chain of large wavelet 
coefficients along a branch of the wavelet tree from a leaf to the root. Leading to tree- 
sparsity where indices in a support form a rooted-connected tree. For such wavelet- 
tree sparsity it is possible to make choices of weights in accordance with the sparsity 
distribution. This motivated the development of the notion of weighted sparsity in [RW15] 
which takes into account prior information on the likelihood that any particular index is 
part of the wavelet tree-sparse support. 

With regards to related works, to the best of our knowledge, performance guaran¬ 
tees for weighted ii minimization in the compressive sensing set-up were independently 
introduced by the works of von Borries et al. [VBMP07], Vaswani and Lu [VLIO], and 
Khajehnejad et al. [KXAH09]. The authors of [VBMP07] showed that exploiting avail¬ 
able support information leads to reduced sampling rates for signals with a sparse discrete 
Fourier transform (DFT); while the authors of [VLIO] also used weighted t'l-minimization 
with zero weights on the support estimate and derived, weaker than £i-minimization, suffi¬ 
cient recovery conditions where a large proportion of the support is known. This work was 
extended to the noisy setting and to compressible signals by [JaclO]. Similarly the authors 
of [KXAH09] proposed the weighted 7i-minimization for the sparse recovery problem with 
known support and analyzed the performance of this method using a Grassmann angle 
approach. An improvement of their result, in the form of a closed form expression for the 
minimum number of measurements, was give in [OKH12], while [FMSY12] further inves¬ 
tigated the stability and robustness of this approach. The authors of [MYll] extended 
the analysis of [KXAH09] to multiple sets. 

From a different angle, attempts to analyze the reweighted ii algorithm introduced 
in [CWB08] also generated interest in weighted £i-minimization in the compressive sensing 
community. The authors of [XKAHIO] attempted to give some rigorous guarantees of im¬ 
provements for the reweighted .^i-minimization over standard 7i-minimization, which was 
lacking in [CWB08], by analyzing a two-step reweighted .^i-minimization scheme. Further 
improvement on the analysis and guarantees of the two-step reweighted 7i-minimization 
was given in [KXAHIO]; while a variant of the reweighted .^i-minimization algorithm was 
proposed in [MY12]. 

Until recently, theoretical guarantees for weighted 7i-minimization largely relied on 
analysis tools of restricted isometry property and null space property that are tailored to 
the analysis of standard 7i-minimization without weights. As such, the recovery guarantees 
for weighted £i-minimization have until recently shown only that weighted 7i-minimization 
cannot do worse than standard .^i-minimization in cases where the weights are aligned with 
the sparsity structure. Recently, the paper [RW15], focusing on function interpolation, 
proved that weighted £i-minimization can provide sample complexity m > Ck\og{N/k) 
for certain classes of structured random matrices where £i-minimization cannot, namely, 
matrices arising from orthonormal systems which are not uniformly bounded in the Lqo 
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norm. To achieve these results, they associate to a set of weights {wj}je[7v] with uij > 1 
the class of what they called weighted s-sparse signals: the weighted sparsity of a vector 
X G is given by ||x||t^^o = Z]{je[iV] ■ x Since ujj > 1, this denotes a smaller 

class of signals than the class of fc-sparse signals. Along with this model class, they in¬ 
troduced a certain weighted null space property to analyze the performance of weighted 
t'l-minimization which, in turn, is strictly weaker than the standard null space property 
required for .^i-minimization. Still, they did not prove that the tools of weighted sparsity 
and weighted null space property can be used to derive sparse recovery guarantees that 
beat the m > Ck\og{N/h) complexity of standard t'l-minimization. A particularly inter¬ 
esting choice of weights suggested in [RW15] and also used in this work is the polynomially 
growing weights, i.e. ojj < j" for a G (0,1/2). This model of weights relates to the ap¬ 
plications of wavelet sparsity and imaging, where essentially one would use polynomially 
growing weights in light of the sparsity distribution. Other works related to the above 
include [PHD14,BBRS15]. 

Later, Mansour and Saab in [MS 14] used a closely related weighted null space property 
and analyzed the performance of weighted .^i-minimization in the two weights regime: 
Ldj = rci > 0 on a subset S C [A^j, and ooj = W 2 = 1 > wi on the complement, S'^. Note 
that this weighting scheme is equivalent to ujj = wi > 1 on 5, and loj = W 2 > wi on 
S^, as rescaling the weighted t'l-minimization objective (WLl) does not change the set of 
minimizers, if the scaling is global on the signal components, i.e., if W 2 /W 1 is constant. 
For a similar reason, without loss of generality one may assume wi = 1. Observe that 
the set S can be interpreted as the set of indices which one has prior reason to believe 
the signal will be localized, and the relative proportion W 2 /VJ 1 on corresponds to a 
confidence in this prior belief. Through the weighted null space property, they provided 
in the noiseless setting, r/ = 0, a reduced sampling complexity for a fixed fc-sparse signal 
X supported on a set 5 of m > C [p^k + n) \og{N/k,) for n = 0 5} U {5 0 5‘^}| < k 

and p = wi/w 2 G [0,1] such that as the estimated support S converges towards the true 
support S, and as the confidence grows as p —)■ 0, the sample complexity improves. Their 
results hold for a fixed support S hence it is referred to as nonuniform, but they showed 
that uniform (holding V5) results will require the “baseline” m > Ck\og{N/k) but this 
bound grows as a function of the error in the support estimate as opposed to the sparsity. 
For more details, see Section 4. 

1.1 Contribution of this work 

The purpose of this work is to provide a unified framework for the sample complexity 
weighted sparse recovery via weighted t'l-minimization, using the tools of weighted sparsity 
and weighted null space property. What is perhaps surprising is that we find that as long 
as the weights in the weighted £i-minimization program grow mildly, namely, if for some 
7 > 0 there is an ordering of the weights satisfying ojj > max{l, \/2y log(j'/s)}, and 
as long as the underlying signal is supported on a set 5 C [N] of weighted cardinality 
td{S) := then a number of linear Gaussian measurements m = 0(s/j) 

suffices for robust recovery of the underlying signal. We show that special cases of this 
main result range from: 
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a) the standard sparse recovery setting where all weights ujj = 1, the unweighted spar¬ 
sity k equals the weighted sparsity s, and m = O {k\og{N/k)) 

b) the setting where the support is known a priori and m = 0{k) Gaussian measure¬ 
ments suffice 

c) the setting of sparse recovery with prior information and using two weights, where 
m depends on how well the weights are aligned with the support set S] hence we 
re-derived the results from [MS 14], but also extended these results to the setting of 
additive noise 

Similar to [MS14] our results are nonuniform that is for a fixed support (which we do 
not need to know a priori), and the weights on S do not all have to be smaller than the 
weights on for our recovery results to hold. 

Underlying our result is a computation for the Gaussian mean width of classes of 
weighted sparse signals. The Gaussian mean width is a notion of the complexity of a set 
and is used to estimate the sample complexity for reconstruction with Gaussian matrices. 
Thus we use it to derive sample complexity bounds for weighted £i-minimization over a set 
of s-weighted sparse signals, which constitutes a subset of the set of unweighted s-sparse 
signals and thus has smaller Gaussian width. This gives a quantitative upper bound (as 
stated in Theorem 4.1) on the sample complexity which depends on how large the weights 
are outside of the support set of x, and ranges from k to k\og{N/k) depending on the 
choice of weights. 

2 Preliminaries 

2.1 Notation &; definitions 

Scalars are denoted by lowercase letters (e.g. /c), vectors by lowercase boldface letter (e.g., 
x), sets by uppercase calligraphic letters (e.g., S) and matrices by uppercase boldface 
letter (e.g. A). The cardinality of a set S is denoted by |5| and [N] N}. Given 

•S C [N], its complement is denoted by 5'^ := [N] \ S and x^ is the restriction of x G 
onto S, i.e. (x^)* = x* if ^ £ 5 and 0 otherwise. 

The £p-norm of a vector x G is defined as 

£i-norm is ||x||^,i ;= 

As in the works of [GWB08, MY12, RW15] 
minimization with a fixed weight ujj > 0 corresponding to each index j G [N], Corre¬ 
sponding to such a set of weights, the paper [RW15] introduced the notion of weighted 
cardinality of a subset S C [N] as a generalization of the standard cardinality: 

Definition 2.1. Fix s G N and a set of weights ujj > 1. A subset S C [N] is said to have 
weighted cardinality s if 

^(‘ 5 ) •= 

i&s 


l^llp (^^=1 i while the weighted 

we consider the setting of weighted £i- 


( 1 ) 
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Correspondingly, the weighted sparsity of a signal x G , denoted by ||x||t^^o = 
^j- Note that for constant weights uij = 1 , the weighted cardinality of a subset 
coincides with the standard cardinality; while the weighted sparsity of a signal coincides 
with the standard sparsity. Otherwise, the set of subsets of [N] of weighted cardinality s 
constitutes a strict subset of those subsets of cardinality s. In turn, the set of s-weighted 
sparse signals, or those x G supported on a subset S of weighted cardinality io{S) < s, 
denotes a strict subset of s-sparse signals. 

3 Weighted sparse recovery with noisy Gaussian measure¬ 
ments 

Our main results on the sample complexity of weighted sparse approximation using Gaus¬ 
sian matrices is based on a calculation for the Gaussian mean width of the set of weighted 
sparse signals. 

In the ensuing analysis we will use the following nonuniform version of the uniform 
weighted robust null space property (cj-RNSP) adapted from Definition 4.1 in [RW15]. 

Definition 3.1 (Nonuniform cj-RNSP, [RW15]). Given a weight vector u, a matrix A G 
satisfy the u-RNSP of order s relative to a fixed set , S C [N], with 
Ld{S) < s and parameters p G (0,1] and t > 0 if 

||z5lU,i < p||z5‘=IU,i + 7 '||Az|| 2 , for all z G M^. (2) 

The standard RNSP where the is replaced by the ii in (2) above is a sufficient 
condition for robust sparse recovery with £i-minimization (LI), see [FR13] and refer¬ 
ences therein. Similarly, a;-RNSP is also a sufficient condition for robust weighted ii- 
minimization (WLl), see [RW15,MS14]. 

Our derivation relies on the work of Ghandrasekaran et al. [GRPW12], where sufficient 
conditions for robust recovery are provided based on the tangent cone of a convex function 
at the signal to be recovered, see also [Trol4]. The tangent cone (also known as the descent 
cone) of the £aj,i-iiorm is defined as follows. 

7I;(x) := cone{u - X ; u G ||u||^^i < ||x||t,j 4 } , 

where “cone” refers to the conic hull of the indicated set. The set 7L(x) consists of the 
directions from x, which do not increase the value of ||x||tj^i. 

However, we use the approach used in [MS 14] where instead of using the tangent cone 
defined above, we use the convex cone of the following set, which contains the set of vectors 
that violate the u;-RNSP. 

- 2 ^ 0 ;,5 := {z G : ||z 5 ||^,i > ||z 5 c||^^i} , (3) 

for S C [N] with cv{S) < s. 

Using this set, we now provide a simple lemma concerning the robust recovery of 
weighted sparse signals. 
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Lemma 3.2. Consider a vector of weights u > 1, and a signal x G with an unknown 
support set S C [A^] of weighted cardinality (jj{S) < s. Let A G 5e a measurement 

matrix, and assume that noisy measurements are taken, y = Ax + e with ||e ||2 < ry. Fix 
( = ((uj, s) > 0 such that 

inf II Az ||2 > C • (4) 

zs^;j,s,||z||2=i 

Then any solution x of (WLl) satisfies 

||x - x ||2 < 2r//C , and ||x - x||^_i < 4?7\/s/C. (5) 

Proof. By the definition of x in (WLl), we have ||x||t^^i < ||x||t^^i. Recalling that x = x^, 
this implies 

||x5c||^^i < ||x||^,i - Psll^p = llx^ll^p - ||X5||^P < ||(x - x)5||^,i, (6) 

that is, that (x — x) G We now apply the triangle inequality to get 

II A(x - x )||2 < II Ax - y ||2 + II Ax - y ||2 < 2r/. (7) 


By (4) we have ||A(x —x )||2 > C||x —x|| 2 - This combined with (7) gives the £ 2 -iiorm error 
bound in (5). To obtain the weighted ^i-norm error bound, note by Cauchy-Schwarz that 

||(x-X)5||^,i < l'^u]\\{x-x)s \\2 < 2r]y/~s/C. (8) 

]l j&s 

As (x — x) G Zi^^s, it follows that 


< ||(x - x)5c||a;,l + ||(x - x) 5 ||^,l < 2||(x - x)5||a;,l < 4:r]^/s/C■ 


Thus concluding the proof. 


□ 


3.1 Gordon’s escape through mesh theorem 

According to (4), successful recovery of a weighted sparse signal is achieved, when the 
minimal gain of the measurement matrix over intersection of the sphere and the set Z^^s 
is greater than some positive constant. For Gaussian matrices the probability of this event 
can be estimated by Gordon’s escape through mesh theorem [Gor88], see also [Theorem 
9.21, [FR13]]. We begin by defining the Gaussian width of a set, which plays a key role 
in the proof of the theorem and our analysis. 

Definition 3.3. Given a standard Gaussian vector, g G with gi ~ AA(0,1), the Gaus¬ 
sian width of a set X C M™, denoted as i{X), is defined as 

i{X) := Esup(g,x). 

xSA” 


( 9 ) 
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Theorem 3.4 (Gordon’s escape through mesh, [Gor88]). Consider an m x N matrix A 
i.i.d 1 

with Qij ~ A/ (0,1); a standard Gaussian random vector g G and a set X C 

For t > 0 we have 


( 10 ) 

We will also require the following standard result about the mean length of a standard 
Gaussian vector, g G M™'. 


inf 

xSA” 


|Ax|| 2 <E||g||2-£(A) 


E||g||2 


v/2r((m + l)/2) 

r (m/2) 


and < E||g ||2 < Vm , 

ym + 1 


where F is the gamma function. 


( 11 ) 


3.2 Gaussian mean width for the set of weighted sparse signals 

To apply Theorem 3.4, we need to estimate I (2iuj,s G , which is given in the following 

lemma. 


Lemma 3.5. Fix weights u)j > 1 and consider a subset 5 C [A] with u){S) < s and 
|5| < /c < s. Let the set be as in (3). Then 


fiZ^sFS^-^) <Vk+mi 

^ ' h>0 


h\/s + 



( 12 ) 


Proof. We start by replacing the sup by the max in the definition of i (Zi^^s G S-^ in 
(9), since Zi^^s G 5"^“^ is compact. Observe that 


max 


(z>g) 


N 


max 

z6.E„,sn5^-in{z 


: zj>0} 


i=i 


Define the convex cone 

Zuj,s ■= Z^^s G {z : Zj > 0} (13) 

and its dual, denoted as Zf^g, is defined; 

; (u,z) > 0 Vz G ^^, 5 } • (14) 

We apply duality to bound the Gaussian width, f {Z^^s G as follows 

f {Zuj,s F S^~^) =E ^max (z,g) (15) 

ze.z„.5ns^-i 

<E mm ||u + g|| 2 , (16) 

s 
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where g has components \gj\, j G [A^]. A linear program in maximization linear form 

max (c,x) s.t. Ax < b, x > 0 (17) 

like in (15), referred to as the primal linear program, has its dual linear program 

min (b, y) s.t. A^y > c, y > 0 (18) 

as in (16). Bound (16) is due to the weak duality principle, which states that the optimum 
of the dual is an upper bound to the optimum of the primal. For details about duality 
theory, see Appendix B.5 of [FR13]. 

To use the above bound we need to characterize the dual cone, 2*^, as follows for 
instance. 

Proposition 3.6. The dual cone of satisfies 

^ ;= y {u G : Uj = hujj for j G S, Uj > -hujj for j G 5'"} . (19) 

h>0 


Proof. For any z G Z^^^S: any u G and any /i > 0 we have 


(u, z) = UjZj + ^ UjZj >hY^ ujjZj - A ^ UjZj 

jes j£S<^ jes j£S‘^ 

= h (llz^llo;,! - ||z5c||^_l) > 0. 


( 20 ) 


The first equality follows from the definition of (•,•), while the first inequality follows from 
the definition of The second equality is due to the definition of || • while the 

last inequality is due to the definition of Z^^^g. This shows that u 


Therefore, using (16) with Proposition 3.6 we bound i {Z^^^s Fl ^) as follows. 


^(- 2 ^ 1.^,5 715" ^) < E min ||u + g ||2 


<E min Iju^c + g5c||2 + E||v5 + g5||2 

1/2 

<E min + E||v5||2 + E||g5||2 

" \jes- 


1/2 




Ui 


T h^/ufiS) -\~ ^/k . 


( 21 ) 

( 22 ) 

(23) 


(24) 


Inequality (22) and (23) are due to the application of the triangle inequality and are true 
for any v G Vaj ,5 and h > 0. The first term in (24) is due to the fact that uj = hiOj for 
j G S and the definition of the weighted sparsity, the second term comes from the upper 
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bound in (11). Next we upper bound the first term of (24) as follows. 

1/2 / X 1/2 




Ui 


<e( 

U'65= 


< I E 


1/2 

, (25) 


where gj is a component of the standard Gaussian vector g, and Shu)^{-) is the soft- 
thresholding operator, 

S\{x) := sign(x) [max(0, |x| — A)] . (26) 

The last inequality in (25) comes from the application of Holder’s inequality. We use 
the symmetry of Gaussian random variable g and the soft-thresholding operation with a 
change of variable to compute E5 a( 5')^, as derived in Section 9.2 of [FR13]: 


1 poo r\ poo 

^Sxigf = J Sx{ufe~^^^‘^du = J Sx{ufe~'^^/‘^du 

Using the bound we can upper bound KSxig)"^ further from (28), 


(27) 

(28) 


^Sx{g? < ^ yXA-2e-AV2 . 

Finally, we upper bound the right hand side (RHS) expression of (25) as 


(29) 


1/2 


E {gj 


< I w — 

Tie 


E 


£-'■‘"1/2 


1/2 


o 

165 “ 1 


We have arrived at an /i-dependent bound for the Gaussian mean width (24). Taking the 
infimum of the bound over /i > 0 provides the lemma. □ 

We now apply the bound in Lemma 3.5 to derive a general result on the sample 
complexity of weighted sparse recovery using Gaussian random measurements. 


Lemma 3.7. Fix weights ojj > 1, and consider a signal x G supported on S C [N"] 

with |5| < k and u}{S) = J2jes ^ j^mxTV ^ AA(0,1), and fix C > 0- 

Assume that noisy measurements are taken, y = Ax -|- e with ||e ||2 < g. If, for some 

Q <5 <1, 


m 


\Jm + l 


> y/k + i 2log ( -] -F C + inf 


/ 


h>0 


hx/~s -|- 


2 


i/2'y 


(30) 
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Then with probability exceeding 1 — 5, any solution x of (WLl) satisfies 

||x — x ||2 < 2r//(‘, and ||x — < 4A/sr//C • (31) 

The proof of Lemma 3.7 in light of Lemma 3.5 is by now standard, see for example 
Theorem 3 of [KRZ14], or more generally in [CRPW12]. We therefore skip the details of 
the proof and provide only a sketch. 

Sketch of proof of Lemma 3. 7. For t > 0, we want to determine 

pf inf \\Az \\2 <E\\g\\ 2 -i(Z^,sn S^-^)-C-t] (32) 

Vze.2i.;,sns^-i / 

We have a lower bound of IE||g ||2 and an upper bound of i n S^~^) in (11) and 
(12) respectively, thus the task remains to choose an appropriate value for t so that we 
have 

E||g||2-^(^^,5n5'^-^) -C-t>o. 

Setting t = y^ 2 log(l/(5) upper bounds the probability (32) by 5, leading to the required 
sampling bound (30) with probability at least 1 — 5 since 

p( inf ||Az ||2 > C ) > IP ( inf || Az ||2 > E||g ||2 - (E:^,5 n t) 

>1-5, 


This concludes the proof. □ 

4 Sample complexity for weighted sparse approximation 

With Lemmas 3.5 and 3.7 in hand, we are now ready to state the main result: under 
very mild growth conditions on the weights loj, the sample complexity of weighted sparse 
recovery given in Lemma 3.7 is linear with respect to the sparsity of the signal. 

Theorem 4.1. Fix weights Uj > 1, and suppose that y > 0 is such that the weights 

u}j > max |l, y^ 27 log(j7s)| , Vj G [N]. Fix A G ^rnxN ^ M{0, 1) and C > 0. 

Consider a signal x G supported on S C [A] with |5| < k and uj{S) = ^ 

Assume that noisy measurements are taken, y = Ax + e with ||e ||2 < rj. Suppose, for 
some 0 < (5 < 1, 


Then with probability exceeding 1 — 5, any solution x of (WLl) satisfies 
||x — x ||2 < 277/(), and ||x — x ||^7 < 4\/s7/C • 


( 34 ) 
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This theorem provides a sharp bound on the sample complexity for weighted sparse approx¬ 
imation in terms of how well the weights used in the weighted £i reconstruction program 
align with the signal support. The more that the support of S coincides with a subset of 
indices having small relative weight (corresponding to smaller weighted sparsity s), and 
the more that the complementary support coincides with indices having larger weight 
(corresponding to larger value of 7 ), the smaller the necessary number of measurements 
m to recover a signal supported on S. Note that the opposite statement is also true: if S 
is supported on a set where the weights are large, then the weighted sparsity s becomes 
large and the sample complexity of weighted sparse approximation becomes higher than 
that for t'l-minimization with uniform weights. 

We can simplify the bound on the number of measurements appearing in (33): using the 
property that for any positive numbers A,B,C and D, 

Vd > VA + Vb + VC =y D >2A + 2B + 2C . (35) 

and ignoring the terms depending on the robustness and confidence parameters C and 5, 
we have the sampling rate of 

m = O (s/ 7 ). (36) 

Indeed, Theorem 4.1 recovers several known results on the sample complexity of sparse 
approximation in special cases (up to a universal constant), as we now outline. 

• Uniform weights. In standard £i-minimization, the weights are all ujj = 1. This 
is a special case of Theorem 4.1 with s = k and 

1 

^ “ 2log{N/k)’ 

thus recovering the known sample complexity m = O {klog{N/k)) for unweighted 
sparse recovery, [AR14, CR06, CRPW12]. 

• Polynomially growing weights. In [RW15], the authors considered the case of 
polynomially growing weights in an application to smooth function interpolation. 
While they did not focus on reduced sample complexity for weights as such, they 
did show that orthonormal function systems which are not uniformly bounded - 
and thus do not satisfy the requirements for unweighted sparse recovery at sample 
complexity m = 0{klog{N/k)) - nevertheless can recover weighted sparse signals at 
this sample complexity with weighted .^i-minimization. It was however mentioned 
in a remark in [RW15] that for weights satisfying polynomial growth, ujj > j", the 
set of s-weighted sparse signals can be reconstructed via weighted ^i-minimization 
using m = 0(a“^s log(s)) Gaussian measurements. Since j" > log(j“) > alog(j'/s). 
Theorem 4.1 implies that in the setting of polynomially growing weights ujj > j“, 
the sample complexity required to recover a weighted s-sparse signal scales no worse 
than m = 0{s/a). 
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• Sparse recovery with prior support estimates. In this setting, one a priori 
splits the support set [A^] into two subsets S and S'^: S, which one has reason 
to believe includes the support of the underlying signals of interest, one places a 
smaller weight tci > 1. On 5'^, one places a larger weight of W 2 > wi. The sample 
complexity thus depends on how well the prior support estimate S agrees with the 
true underlying support S and on the confidence W 2 /W 1 in this belief. In terms of 
weighted sparse recovery, the more disjoint are the true and estimated support sets 
S and S, the larger will be the weighted sparsity of the signal supported on S. 

Normalize without loss of generality so that wi = 1 and W 2 '■= l/p > 1. Suppose 
that X is supported on a set S of size |5| = k. Then Wj = 1 for j G 5 n 5, and 
cuj = l/p > 1 for j G 5 n S^. It follows that the weighted cardinality of the support 

is 

s = u;(5) = |5n5|+p-2|5n5‘=| (37) 

At the same time, Uj = I for j G 5“^ n 5 and loj = \jp > I for j G 5*^ H S^. 
Assuming without loss that the indices in S'^ are ordered according to the increasing 
arrangement of their weights, we find that for j G 5“^, 

(jj > max { 1,^27 log(j7s)} , with 7 = min 11, J . (38) 

Applying Theorem 4.1 gives thus the sample complexity for sparse recovery with 
prior support estimates: 

m = O = O [sp^ \og{N/s) + s) 

= O log(A^/s)+ s) . (39) 

As in [MSI4], fix parameters a, j3 G [0,1], and suppose |5| = /3|5| = jdk, and suppose 
that |5 n 5| = a|5|. Then if 2p^ \og{N/s) > I (=> p > —> 0 if iV / s), the 

dominant term in the sampling bound in (39) is bounded above by 

m = O k + n) \og{N/k)) , (40) 

where n = |5'^n5| + |5n5‘^| represents the mismatch between the true and estimated 
supports. This aligns with the sample complexity for sparse approximation with 
prior belief from [MSI4]. If p —)■ 0, which necessitates p < -j===j=^ implying 

2p^\og{N/s) < I, and as n —)• 0, the sample complexity goes to s. That is as the 
weights W 2 are increased relative to the weights rci, and as the mismatch parameter 
n —)■ 0 , then m —)■ 0{s). 

Moreover, since our results are for the noisy setting, in addition to recovering the 
results of [MS 14], we generalize those results which were shown only in the noiseless 
setting. 
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• Known support. In case x is supported on 5 = 5 of size |5| = /c = s esti¬ 
mated exactly, then if is sufficiently small such that p^\og{N/k) < 1, the sample 
complexity reduces to 

m = 0{k), 

recovering the known sample complexity of solving for a /c-sparse signal with known 
support from Gaussian measurements. 


Proof of Theorem 4-1- Suppose without loss of generality that = [N] and that the 
weights are chosen to be ooj = max{l, log(j'/s)} for each j G [N], Then 


^^ < 
h^o:] - - 


^ 2g h? 12^ j < s • 

h~‘^, j > s ■ 


Set h = \/2jy. It follows that 


-< 

j6cS‘= 1 


N 

-}dbP:l2 

<-lU 

N 

E 

e ^ 

h^u?- 

-1/(27) + 

i=i 

j 

\ 




^ 37s _ 
\ - e 

1/(27) < ^ 



~ 2 

“ 2 e 


-2 


(41) 


(42) 


The result is then arrived at by substituting this value of h and applying Theorem 3.7. 
This completes the proof. □ 


5 Numerical simulations 

5.1 Weighted sparse recovery 

To perform weighted sparse recovery we generate signals from a distribution we refer to 
as the weighted sparse signal model. Signals generated from this distribution are weighted 
s-sparse in expectation. To be precise, the probability for an index to be in the support 
of the signal is proportional to the reciprocal of the square of the weights assigned to that 
index. In particular, let’s consider weighted s-sparse signals with support S, i.e. oj{S) < s, 
we draw indices of the signal with probability of being nonzero as 

Prob (xj 0) = s/N ■ l/cjj , (43) 

where s and N are the weighted sparsity and the ambient dimension of the signal, x G 
respectively. The weights LOj will then be used in the weighted .^i-minimization (WLl) 
recovery of x. It is not hard to show that the weighted sparsity in expectation is s as 
shown below. 

N N 

E [u}{S)] = Prob {xj / 0) • =Y • 

i=i i=i 


( 44 ) 
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In the experiments presented below, we draw signals of dimension N = 500 from the 
above model, where the nonzero values are randomly generated as scaled Gaussian random 
variables without any normalization, and draw m x N sensing matrices with entries i.i.d. 
from a normal Gaussian. Noisy measurements are drawn with Gaussian noise ||e ||2 = 10“®. 
Then we run simulations with 50 realizations for each problem instance {s,m,N). 

Here we set out to compare sample complexities of weighted sparse recovery using 
weighted .^i-minimization and standard sparse recovery using unweighted ^i-minimization 
for signals generated from the weighted sparse signal model. We vary the number of mea¬ 
surements m such that m/N G [0.05,0.5]; and fix weighted sparsity of the supp(x), S, 
such that uj(S)/m = s/m G [ q osiV ’ • Then we record k as the largest |5| for a given 
s. We considered two regimes of weights: 1) polynomially growing weights, i.e. ojj = 
for 6 > 0 and j = (or more generally uij = vr(j)® for a permutation vr); and 

2) uniformly random generated weights. In both regimes we consider a reconstruction 
successful if the recovery error in the ^ 2 -iiorm is below 10“^ and a failure otherwise. Then 
we compute the empirical probabilities as the ratio of the number of successful reconstruc¬ 
tions to the number of realizations. We present below sample complexity comparisons in 
these two regimes of weights using the phase transition framework in the phase space of 
(s/m, m/N). Note that in all the figures we standardized the values of s/m in such a way 
that the standardized s/m is between 0 and 1 for fair comparison. 

• Polynomially growing weights. Precisely, we assign weights that grow polynomi¬ 
ally proportional to ooj = . Fig. 1 shows surface plots of empirical probabilities 

for signals drawn from the distribution of weighted sparse signals using polynomially 
growing weights. The color bars depict the probabilities of recovering “close enough” 
approximation x to the true x. By having larger area of yellow (i.e. with probability 
one), weighted sparse recovery using (WLl), in the left panel, outperforms generic 
sparse recovery using (LI), in the right panel. Note that weighted t'l-minimization 



0.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 

mjN mjN 

Figure 1: Surface plots depicting phase transitions Left panel: Weighted £i-minimization. Right 
panel: unweighted £i-minimization. 

has a seemingly high phase transition. This is because the generic sparsities are 
small for the pre-specified weighted sparsities, which depends on the magnitude of 
the weights. This can be seen in the left panel of Fig. 2 where the largest k/m is 
roughly 0.63. Furthermore, we computed the average empirical weighted sparsity in 
our simulations and as can be seen in the left panel of Fig. 2, these average weighted 
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sparsities are very close to the pre-specified weighted sparsities. Thus confirming 
our claim that the signals we generate are weighted sparse in expectation. 
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Figure 2: Sparsity ratios k/m versus standardized weighted sparsity ratios s/m] standardized 
such that s/m S [0,1]. Left panel: Polynomially growing weights. Right panel: Random weights. 


• Random weights. Precisely, we assign weights that are uniformly randomly gen¬ 
erated and normalized such that minjOjj = 1. Similarly, Fig. 3 shows surface plots 
of empirical probabilities for signals drawn from the distribution of weighted sparse 
signals with these random weights. Again, by having larger area of yellow, weighted 
sparse recovery using (WLl), in the left panel, outperforms generic sparse recovery 
using (LI), in the middle panel. 



0.1 0.2 0.3 0.4 

m/N 



Figure 3: Surface plots depicting phase transitions Left panel: Weighted £i-minimization. Right 
panel: unweighted £i-minimization. 


Note that unweighted -minimization has a relatively lower phase transition curve for £ 1 - 
minimization with random weights (of relatively smaller weights) than the polynomially 
growing weights in the right panel of Fig. 1. This is because the generic sparsities are 
larger for the pre-specified weighted sparsities, see the right panel of Fig. 2. The average 
weighted sparsities are very close to the pre-specified weighted sparsities, see again the 
right panel of Fig. 2. Thus again confirming our claim that the signals we generate are 
weighted sparse in expectation. 
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5.2 Sparse recovery with prior support estimates 

In this second set of experiments we show how incorporating additional information about 
the support of the signal affects sparse recovery using (weighted) t'l-minimization. The 
setup is that for a support S we know an estimate of the support we denote as S. We con¬ 
sidered support estimates S of different degrees of overlap with the support S. Technically, 
the overlap is parameterized by 0 < a,/3 < 1 such that |5| = /3|5|, |5n5| = a|5| = a/?|5|; 
but for simplicity in these experiments we fix /3 = 1 and vary a with a = {0,1/3, 2/3,1}. 
Recall that a = 0 means that the estimated support is not aligned with the true support, 
while a = 1 indicates perfect alignment. 

We assign weights such that uoj = 1 for j G 5 and iOj = W 2 = 1/w > 1 for j £ S‘^ 
with w = 1 — a as, was prescribed in [MS 14] as an optimal choice of weights. The problem 
dimensions are the same as in the previous section except that we pre-specify s/m G 
[T o 05 V ~l ’ ^2(1 “ ®/ 5 )] • the definition of the weighted sparsity s we can compute 

the generic sparsity k = \• 

• Phase transitions. Our theoretical results show that using good weights in the 
weighted £i-minimization reconstruction, we can recover sparse signals on support 
sets aligned with the weights using fewer measurements m. Fig. 4 shows contour 
plots of empirical probabilities (the ratio of the number of reconstructions with errors 
below a given tolerance to the number of realizations) of 50% and 95% respectively. 
The curves depict the sampling rates that give 50% or more (respectively 95% or 
more) chance of getting a “close enough” approximation x to the true x. 




Figure 4: Contour plots of phase transitions. Left panel: 50% contour curves. Right panel: 95% 
contour lines. 

These results show that weighted sparse recovery using (WLl), outperforms generic 
sparse recovery using (LI), if the support estimate is significantly accurate. When 
a = 1 we almost have perfect recovery using weighted l\ and even when 0 = 2/?) 
weighted fi does significantly better than unweighted ii. This confirms the results 
of [MS14] in the noiseless setting, which guarantees improvement when a > 50%. 
We see not much improvement of weighted li over unweighted fi when a = 1/3 and 
when a = 0 we get no improvement of weighted £1 over unweighted ii as one would 
expected. 
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Furthermore, the left panel of Fig. 5 shows how the weighted sparsity varies with 
generic sparsity. In particular, we see an increase in the weighted sparsity as a 
increases from 0 to 1/3 to 2/3 but due to the optimal choice of the weights of 
W 2 = 1 — a, a = 0 and a = 1 have the same weighted sparsity versus generic 
sparsity profile. The two curves coincide thats why only three curves appeared in 
the right panel plot. Alternatively, the result depicted in Fig. 4 can be zoomed into 
via a slice through the phase transition curves for a fixed m, as shown in the right 
panel of Fig. 5 for m = A^/4. 



0.2 0.4 0.6 0.8 1 


kjm 



Figure 5: Left panel Sparsity ratios k/m versus weighted sparsity ratios s/m Right panel Re¬ 
covery probabilities as a function of the weighted sparsity for a fixed m = N/A. 


• Recovery errors. We also compare recovery errors of the two approaches men¬ 
tioned above in the two weight setting. We compute the errors for a fixed number of 
measurements m = A^/4 and pre-specified s/m G [T o 05A'' ~I ’ + ^ 2(1 “ ct/3)] ■ We 

denote the recovered signal as Xaig-, where alg is either or li, referring to the 
recovery algorithm used, i.e. (WLl) or (LI) respectively. We present relative errors 
for varying a G {0,1/3, 2/3,1}. 




Figure 6: Relative reconstruction errors. Left panel ^ 2 -norm Right panel 


The left panel of Fig. 6 shows the 1 ' 2 -norm relative errors, i.e. := while 

II 5C|| 2 

the right panel of Fig. 6 shows the i-norm relative errors, i.e. := _ 

’ II ^112 
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As implied by the results discussed above, Fig. 6 confirms that sparse recovery using 
(WLl) get smaller £2 (respectively iujg) errors than generic sparse recovery using (LI) as 
a increases. 


6 Conclusion 

This manuscript unifies, strengthens, and extends previous weighted fi recovery bounds, 
using the tools of weighted sparsity and weighted null space property. In particular, 
we provided a new general sample complexity bound for weighted sparse recovery with 
Gaussian matrices using weighted .^i-minimization, by deriving a bound on the Gaussian 
mean width for nonuniform robust weighted sparse recovery. 

Special cases of our results include the case of polynomially-growing weights and the 
case of prior information in the form of two weights, whereby we recover the results 
of [MS14]. Our main sample complexity result unifies and extends these special cases to 
the general setting of an arbitrary weight distribution Uj > 1 by framing the result in 
terms of weighted sparse recovery bounds, and moreover generalizes these bounds to the 
noisy setting. 

Several questions remain open for future work. On the one hand, our results are for 
Gaussian measurement matrices, and it would be interesting to explore to what extent 
these results hold for more structured random measurement ensembles. Finally, it is 
interesting to see whether our theory can be used to derive theoretical guarantees for re¬ 
weighted £i-minimization, which is an iterative application of weighted t'l-minimization 
where weights are learned in conjunction with the underlying sparse solution. 


Acknowledgements 

The work was funded, in part, by an NSF CAREER Grant and AFOSR Young Investigator 
Award. The authors would like to thank Soledad Villar, for reading the paper and giving 
valuable feedback that improved the manuscript, and Anastasios Kyrillidis, for providing 
feedback on the experiments. We would also like to thank the anonymous referees. 


References 

[AR14] U. Ayaz and H. Rauhut. Nonuniform sparse recovery with subgaussian ma¬ 
trices. Electronic Transactions on Numerical Analysis, 41:167-178, 2014. 

[BBRS15] J.-L. Bouchot, B. Bykowski, H. Rauhut, and C. Schwab. Compressed sensing 
Petrov-Galerkin approximations for parametric PDEs. 2015. 

[CR06] E. J. Candes and J. Romberg. Quantitative robust uncertainty principles and 
optimally sparse decompositions. Foundations of Computational Mathematics, 
6(2):227-254, 2006. 



The sample complexity of weighted sparse approximation 


20 


[CRPW12] 

[CRT06] 

[CWB08] 

[DonOG] 

[FMSY12] 

[FR13] 

[Gor88] 

[JaclO] 

[KRZ14] 

[KXAH09] 

[KXAHIO] 

[MS 14] 
[MYll] 


V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky. The convex 
geometry of linear inverse problems. Foundations of Computational mathe¬ 
matics, 12(6);805-849, 2012. 

E. J. Candes, J. Romberg, and T. Tao. Stable signal recovery from incom¬ 
plete and inaccurate measurements. Communications on pure and applied 
mathematics, 59(8): 1207-1223, 2006. 

E. J. Candes, M. B. Wakin, and S. P. Boyd. Enhancing sparsity by reweighted 
ii minimization. Journal of Fourier analysis and applications, 14(5-6):877- 
905, 2008. 

D. L. Donoho. Compressed sensing. IEEE Trans. Inform. Theory, 52(4);1289- 
1306, 2006. 

M. P. Priedlander, H. Mansour, R. Saab, and O. Yilmaz. Recovering compres- 
sively sampled signals using partial support information. Information Theory, 
IEEE Transactions on, 58(2):1122-1134, 2012. 

S. Eoucart and H. Rauhut. A mathematical introduction to compressive sens¬ 
ing. Springer, 2013. 

Y. Cordon. On Milman’s inequality and random subspaces which escape 
through a mesh in n. Springer, 1988. 

L. Jacques. A short note on compressed sensing with partially known signal 
support. Signal Processing, 90(12):3308-3312, 2010. 

M. Kabanava, H. Rauhut, and H. Zhang. Robust analysis ^i-recovery from 
gaussian measurements and total variation minimization. arXiv preprint 
arXiv: 1407 . 7402 , 2014. 

A. M. Khajehnejad, W. Xu, A. S. Avestimehr, and B. Hassibi. Weighted 
ti minimization for sparse recovery with prior information. In Information 
Theory, 2009. ISIT 2009. IEEE International Symposium on, pages 483-487. 
IEEE, 2009. 

A. M. Khajehnejad, W. Xu, A. S. Avestimehr, and B. Hassibi. Improved sparse 
recovery thresholds with two-step reweighted ii minimization. In Information 
Theory Proceedings (ISIT), 2010 IEEE International Symposium on, pages 
1603-1607. IEEE, 2010. 

H. Mansour and R. Saab. Recovery analysis for weighted ^i-minimization 
using a null space property. arXiv preprint arXiv: 1412.1565, 2014. 

H. Mansour and C. Yilmaz. Weighted-£i minimization with multiple weight¬ 
ing sets. In SPIE Optical Engineering J- Applications, pages 813809-813809. 
International Society for Optics and Photonics, 2011. 



The sample complexity of weighted sparse approximation 


21 


[MY12] 

[0KH12] 

[PHD14] 

[RW15] 

[Trol4] 

[VBMP07] 

[VLIO] 

[XKAHIO] 


H. Mansour and O. Yilmaz. Support driven reweighted fi minimization. In 
Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International 
Conference on, pages 3309-3312. IEEE, 2012. 

S. Oymak, M. A. Khajehnejad, and B. Hassibi. Recovery threshold for optimal 
weight ii minimization. In Information Theory Proceedings (ISIT), 2012 IEEE 
International Symposium on, pages 2032-2036. IEEE, 2012. 

J. Peng, J. Hampton, and A. Doostan. A weighted £i-minimization approach 
for sparse polynomial chaos expansions. Journal of Computational Physics, 
267:92-111, 2014. 

H. Rauhut and R. Ward. Interpolation via weighted fi minimization. Applied 
and Computational Harmonic Analysis, 2015. 

J. A. Tropp. Convex recovery of a structured signal from independent random 
linear measurements. arXiv preprint arXiv:1405.1102, 2014. 

R. Von Borries, J. C. Miosso, and C. M. Poles. Compressed sensing using prior 
information. In Computational Advances in Multi-Sensor Adaptive Processing, 
2007. CAMPSAP 2007. 2nd IEEE International Workshop on, pages 121-124. 
IEEE, 2007. 

N. Vaswani and W. Lu. Modified-CS: Modifying compressive sensing for prob¬ 
lems with partially known support. Signal Processing, IEEE Transactions on, 
58(9):4595-4607, 2010. 

W. Xu, M. A. Khajehnejad, A. S. Avestimehr, and B. Hassibi. Breaking 
through the thresholds; an analysis for iterative reweighted fi minimization 
via the grassmann angle framework. In Acoustics Speech and Signal Processing 
(ICASSP), 2010 IEEE International Conference on, pages 5498-5501. IEEE, 
2010. 



